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ABSTRACT 

The  Air  Force  Research  Laboratory’s  Human  Effectiveness  Directorate  supports  research  addressing  human  factors 
associated  with  Unmanned  Aerial  Vehicle  (UAV)  operator  control  stations.  One  research  thrust  focuses  on  determining 
the  value  of  combining  synthetic  vision  data  with  live  camera  video  presented  on  a  UAV  control  station  display. 
Information  is  constructed  from  databases  (e.g.,  terrain,  etc.),  as  well  as  numerous  information  updates  via  networked 
communication  with  other  sources.  This  information  is  overlaid  conformal,  in  real  time,  onto  the  dynamic  camera  video 
image  display  presented  to  operators.  Synthetic  vision  overlay  technology  is  expected  to  improve  operator  situation 
awareness  by  highlighting  elements  of  interest  within  the  video  image.  Secondly,  it  can  assist  the  operator  in  maintaining 
situation  awareness  of  an  environment  if  the  video  datalink  is  temporarily  degraded.  Synthetic  vision  overlays  can  also 
serve  to  facilitate  intuitive  communications  of  spatial  information  between  geographically  separated  users.  This  paper 
discusses  results  from  a  high-fidelity  UAV  simulation  evaluation  of  synthetic  symbology  overlaid  on  a  (simulated)  live 
camera  display.  Specifically,  the  effects  of  different  telemetry  data  update  rates  for  synthetic  visual  data  were  examined 
for  a  representative  sensor  operator  task.  Participants  controlled  the  zoom  and  orientation  of  the  camera  to  find  and 
designate  targets.  The  results  from  both  performance  and  subjective  data  demonstrated  the  potential  benefit  of  an 
overlay  of  synthetic  symbology  for  improving  situation  awareness,  reducing  workload,  and  decreasing  time  required  to 
designate  points  of  interest.  Implications  of  symbology  update  rate  are  discussed,  as  well  as  other  human  factors  issues. 

Keywords:  synthetic  vision,  conformal  overlay,  situation  awareness,  unmanned  aerial  vehicle,  UAV,  update  rate 

1.  BACKGROUND 

The  success  of  unmanned  aerial  vehicles  (UAVs)  in  recent  military  situations  has  led  to  increased  interest  in  their 
capabilities  and  their  application  to  a  variety  of  missions.  Because  there  is  no  onboard  crew,  UAVs  offer  a  number  of 
distinct  advantages  to  traditional,  manned  aircraft,  especially  for  dull,  dirty,  and  dangerous  missions.1  However,  the 
physical  separation  of  the  crew  from  the  aircraft  also  presents  additional  challenges  to  the  effective  design  of  the  UAV 
control  station.  Numerous  human  factors  issues  such  as  system  time  delays,  poor  crew  coordination,  high  workload,  and 
reduced  situational  awareness  may  negatively  affect  mission  performance.2  When  onboard  an  aircraft,  a  pilot  and  crew 
receive  a  rich  supply  of  multi-sensory  information  instantaneously  regarding  their  surrounding  environment.  UAV 
operators,  however,  may  be  limited  to  a  time-delayed,  reduced  stream  of  sensory  feedback  delivered  almost  exclusively 
through  the  visual  channel.  Making  the  information-management  demands  on  the  UAV  operator  even  worse  are  the 
massive  amounts  of  data  now  available  with  net-centric  warfare  doctrine.  Plus,  as  UAV  platforms  become  more 
autonomous,  it  is  envisioned  that  a  single  crew,  or  one  operator,  will  be  tasked  with  multiple  UAVs  simultaneously,  in 
contrast  to  the  current  manning  of  one  or  more  operators  to  control  a  single  UAV.3 

The  sheer  volume  of  critical  disparate  data  and  imagery  pertaining  to  multiple  UAVs,  together  with  the  new  paradigm 
of  network-centric  warfare,  will  place  a  heavy  burden  on  UAV  operators  who  must  rapidly  make  time-critical  decisions 
and  inputs  to  ensure  successful  mission  completion.  Besides  monitoring  the  maneuvering  of  multiple  UAVs 
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and  the  operation  of  their  respective  subsystems,  operators  will  need  to  decipher  sensory  information  and  make 
assessments  and  decisions  concerning  potential  threats.  With  the  complexity  of  UAV  systems  and  mission  dynamics,  it  is 
also  likely  that  the  operator  will  have  to  intervene  with  the  subsystems  of  one  or  more  UAVs  to  handle  contingency 
operations.  In  order  for  UAV  operators  to  rapidly  understand  incoming  data  and  act  on  assets  under  their  control,  more 
sophisticated  enabling  visualization  tools  are  needed  to  promote  real-time  interpretation  of  data  and  increase  situation 
awareness. 

Synthetic  vision  systems4  (also  termed  augmented  reality)  can  potentially  enhance  data  interpretation  and  the  level  of 
operator  situation  awareness,  and  consequently  influence  the  decision  making  capability  and  the  ability  to  make  timely 
responses  to  contingencies.  With  this  technology,  spatially-relevant  information,  constructed  from  databases  (e.g.,  terrain 
elevation,  maps,  photo-imagery,  etc.)  as  well  as  networked  information  sources,  can  be  represented  as  computer 
symbology  and  overlaid  conformal,  in  real  time,  onto  operator  displays.  This  computer-generated  symbology  appears  to 
“co-exist”  with  real  objects  in  the  visual  scene,  highlighting  points  of  interest  to  operators. 

2.  SYNTHETIC  VISION  CONCEPTS  FOR  UAV  OPERATIONS 

2.1  Potential  Applications  for  UAVs 

It  is  anticipated  that  a  synthetic  visual  overlay  technology  could  benefit  UAV  missions.  Perhaps  one  of  the  most 
promising  UAV  application  for  synthetic  symbology  is  to  augment  the  display  showing  video  imagery  from  various 
cameras  mounted  on  the  UAV.  Pilots  use  imagery  from  the  UAV’s  nose  and  gimbal  cameras  to  verify  clear  path  for 
taxi/runway  operations,  scan  for  other  air  traffic  in  the  area,  and  identify  navigational  landmarks  and  potential 
obstructions.  Sensor  operators  use  imagery  from  a  gimbal-mounted  camera  to  conduct  a  wide  variety  of  intelligence, 
surveillance  and  reconnaissance  activities  as  well  as  to  directly  support  combat  operations.  However,  video  imagery 
quality  can  be  compromised  by  narrow  camera  field-of-view,  datalink  degradations,  poor  environmental  conditions  (e.g., 
dawn/dusk/night,  adverse  weather,  variable  clouds),  bandwidth  limitations,  or  a  highly  cluttered  visual  scene  (e.g.,  in 
urban  areas  or  mountainous  terrain). 

A  synthetic  vision  system  can  potentially  enhance  UAV  operator  interpretation  of  data/imagery,  improve  situation 
awareness,  and  ameliorate  negative  video  characteristics.5  It  could  improve  operator  situation  awareness  by  highlighting 
elements  of  interest  within  the  video  image  such  as  threat  locations,  key  landmarks,  emergency  airfields,  etc.  Secondly,  it 
can  assist  the  operator  in  maintaining  situation  awareness  of  an  environment  if  the  video  datalink  is  temporarily 
degraded.  Synthetic  vision  overlays  could  also  serve  to  facilitate  intuitive  communications  of  spatial  information 
between  geographically  separated  users.  The  application  of  a  synthetic  overlay  to  the  video  imagery  display  in  UAV 
stations  is  illustrated  in  Figures  1  and  2.  Figure  1  shows  synthetic  vision  symbology  developed  by  Rapid  Imaging 
Software,  Inc.  (SmartCam3D)  added  to  (simulated)  UAV  video  imagery.  Figure  2  shows  another  synthetic  overlay 
concept,  termed  “picture-in-picture”  whereby  synthetic-generated  imagery  surrounds  the  real  video  imagery  on  the 
display.  This  concept  affords  virtual  expansion  of  the  available  sensor  field-of-view  well  beyond  the  physical  limits  of 
the  camera,  potentially  improving  large  area  situation  awareness. 
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Figure  2:  Picture-in-picture  concept,  with  real  video  imagery  surrounded  by  synthetic-generated  terrain  imagery. 

2.2  Past  research  addressing  application  of  synthetic  vision  systems  to  UAVs 

Presentation  of  a  synthetic  visual  overlay  in  a  conformal  manner  with  sensor  imagery  has  been  demonstrated  in  the 
laboratory  to  reduce  scanning  time,  reduce  the  need  to  mentally  integrate  spatial  information  from  disparate  sources,  and 
facilitate  attentional  management.6  To  date,  application  research  has  primarily  focused  on  how  synthetic  overlays  can  aid 
piloting  tasks  during  manned  flights.  In  fact,  there  is  a  large  body  of  research  showing  the  utility  of  synthetic  vision 
systems  for  both  military  and  civilian  manned  flight  applications.  For  instance,  flight  guidance  overlay  symbology  has 
been  found  to  be  valuable  for  reduced  visibility  conditions,  especially  during  landings.7 

Synthetic  overlay  research  for  UAV  applications  is  scant  in  comparison  to  research  for  manned  applications.  The 
majority  of  efforts  to  date  have  involved  developing  concept  demonstrations  and  acquiring  subjective  data  from  subject- 
matter-experts.  For  instance,  in  an  U.S.  Army  Research  Laboratory  effort,  synthetic  vision  overlay  concepts  are  being 
employed  in  the  development  of  displays  to  support  multi-UAV  control.8  On  the  format,  lines  radiate  from  the  UAV 
symbols  to  show  their  visual  field-of-regard  in  relation  to  themselves  and  the  ground.  Multicolored  lines  on  the  ground 
delineate  political  boundaries.  Ground  targets  of  interest  are  highlighted  with  black  hexahedrons  and  a  north  facing 
arrow  fixed  on  the  screen  provides  geographic  orientation  and  a  situation  awareness  enhancement.  Tests  are  ongoing, 
but  results  to  date  suggest  that  the  ability  to  combine  real  time  video  imagery  with  virtual  databases  will  enhance 
situation  awareness  and  improve  crew  coordination.  Academic  research  in  the  Netherlands  has  also  addressed  the 
feasibility  of  using  synthetic  vision  technology  for  UAV  operator  support.  Using  existing  synthetic  vision  software 
components,  a  UAV  mission  management  system  was  created  to  support  ongoing  research  to  see  if  computer  generated 
imagery  can  increase  situation  awareness,  both  for  the  current  situation  as  well  as  understanding  the  effect  of  changes  on 
future  situations  in  the  mission.9, 10 

In  a  recent  study  conducted  by  the  U.S.  Air  Force  Research  Laboratory,  both  performance  and  subjective  data  were 
obtained  during  the  evaluation  of  three  synthetic  visual  overlay  concepts  for  improving  the  situation  awareness  of  UAV 
sensor  operators.11  Results  indicated  that  the  concepts  showed  promise  for  improving  UAV  operations,  particularly  the 
use  of  virtual  flags  to  highlight  landmarks.  The  overlay  of  virtual  flags  significantly  reduced  landmark  designation  times 
(by  40-58%)  and  served  to  mitigate  the  negative  effect  of  low  camera  slew  accuracy.  These  data  also  did  not  show  any 
cognitive  tunneling  effects.  Attention/cognitive  tunneling  occurs  when  the  operator  becomes  fixated  on  the  synthetic  cue 
itself  (or  objects  to  which  attention  is  directed  by  the  synthetic  cue)  to  such  an  extent  that  other,  un-cued  events  known  to 
be  of  importance  are  not  attended  to.12  In  the  case  of  UAVs,  this  may  result  in  the  failure  to  detect  unexpected,  high 
priority  objects  . 

Although  the  data  from  this  recent  study  did  not  show  any  cognitive  tunneling  effects,  it  may  be  that  the  highly  salient 
unexpected  target  utilized  washed  out  any  tunneling  effects.  Follow-on  research  is  needed  to  further  examine  the 
potential  for  cognitive  tunneling,  in  addition  to  a  myriad  of  other  human  factors  issues  in  regards  to  synthetic  vision 
systems  (see  2.3). 
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2.3  Human  factors  issues  with  UAV  synthetic  vision  systems 

There  are  numerous  human  factors  issues  to  consider  in  the  application  of  synthetic  vision  systems  to  UAVs.13  For 
instance,  there  are  multiple  issues  on  how  the  synthetic  vision  symbology  should  be  presented  in  terms  of  optimizing  each 
symbology  element  and  determining  if  the  symbology  should  appear  on  a  separate  display  versus  an  overlay  and,  if  the 
latter,  if  it  should  be  superimposed  versus  “scene-linked.”  Applications  also  need  to  take  into  account  the  potentially 
negative  effects  of  information  clutter  by  only  including  elements  that  will  benefit  the  operators’  situation  awareness  and 
performance,  and  employing  design  features  that  minimize  clutter  effects.  Information  view  management  technologies 
can  be  used  to  optimize  the  visibility  of  the  synthetic  vision  symbology  for  a  variety  of  viewing  situations,  as  well  as 
indicate  the  criticality,  urgency,  reliability,  and  timeliness  of  information  depicted  by  elements.  Blending  techniques  may 
be  appropriate,  whereby  the  transparency  of  the  synthetic  symbology  is  manipulated.  The  performance  of  the  synthetic 
vision  system  (registration  errors  and  time  delays)  can  also  impact  the  utility  of  the  symbology.  Besides  displays  issues, 
there  are  also  control  issues:  what  control  interfaces  will  be  used  by  operators  to  modify  the  symbology  and  how 
distributed  network  collaborative  communication  of  synthetic  information  will  be  implemented.  In  sum,  evaluations 
(usability,  simulation,  and  flight  test)  are  needed  to  optimize  how  synthetic  information  is  implemented  in  UAV 
applications  and  to  confirm  that  the  synthetic  vision  system  will  benefit  UAV  operations  and  result  in  increased  mission 
effectiveness.  This  paper  reports  the  results  of  a  recent  simulation  assessment  of  update  rate  on  synthetic  symbology 
utility  for  a  UAV  operator  task. 


3.  METHOD 


3.1  Objective 

The  rate  at  which  data  on  the  (simulated)  UAV’s  position  in  space  is  fed  to  the  software  that  generates  the  synthetic 
symbology  was  manipulated  to  determine  its  effect,  if  any,  on  the  utility  of  the  synthetic  symbology  presented  for  task 
completion.  Update  rate  is  a  factor  of  interest  because  it,  in  turn,  contributes  to  the  appearance/behavior  of  the  synthetic 
symbology,  as  well  as  registration  errors.  At  higher  rates,  the  synthetic  symbology  appears  smooth,  not  jumpy,  and  with 
no  delay.  At  slower  rates,  the  symbology  stutters  or  jumps  and  it  may  be  possible  to  detect  delays.  Even  if  the 
symbology  appears  to  jump,  etc.,  it  is  possible  that  this  is  not  distracting  to  operators  and/or  may  not  impact  task 
performance.  Another  possibility  is  that  the  effect  of  update  rate  may  differ  as  a  function  of  task  type.  For  tasks  where 
the  synthetic  symbology  is  used  to  highlight  objects  in  the  real  world,  it  is  especially  important  for  the  synthetic  and  real 
worlds  to  be  properly  aligned  (i.e.,  registered)  with  respect  to  each  other  on  the  display,  or  the  illusion  that  the  two 
worlds  coexist  will  be  compromised.14  Slower  update  rates  may  contribute  to  the  registration  error,  making  it  more 
difficult  to  discern  what  real  world  element  is  being  highlighted  by  a  synthetic  cue.  In  contrast,  for  tasks  in  which  the 
operator  needs  to  retrieve  information  from  the  synthetic  symbology  (for  example,  enter  text  displayed  on  a  synthetic 
flag),  the  stability  of  the  symbology  may  be  more  important  than  how  recently  it  was  refreshed.  In  this  case,  update  rates 
very  low  and  very  high  may  be  better  than  update  rates  in  a  middle  range  that  result  in  the  symbology  jumping  around. 

For  the  present  experiment,  seven  different  telemetry  update  rates  were  examined:  .5,  1,  2,  4,  6,  10,  and  24  Hz. 

These  update  rates  were  chosen  to  obtain  task  performance  data  with  several  candidate  operational  rates  for  a  variety  of 
UAVs.  Another  objective  was  to  explore  if  there  is  a  point  where  task  performance  starts  to  degrade  in  the  relationship 
between  telemetry  update  rate  and  the  delay  in  the  presentation  of  synthetic  information.  This  relationship  can  be 
described  as  a  sharp  curve  with  a  2  second  delay  at  the  .5  telemetry  update  rate,  dropping  to  a  0.  25  second  delay  by  4 
Hz. 


3.2  Simulation  environment 

The  UAV  sensor  operator  workstation  (Figure  3)  had  head-level  and  upper  17”  color  displays,  as  well  as  two  10” 
head-down  color  displays  (HDDs).  During  the  experiment,  participants  focused  on  the  head-level  display  presenting 
simulated  video  imagery  from  the  UAV’s  gimbaled  camera,  along  with  head-up  display  (HUD)  sensor  symbology.  The 
simulated  imagery  was  a  realistic  database  model  of  southern  Nevada.  For  the  present  experiment,  the  upper  display  was 
covered,  such  that  participants  could  not  see  its  map  with  symbology  identifying  current  UAV  location,  flight  corridor, 
and  landmark  locations.  This  was  done  to  control  the  source  of  information  for  all  participants  -  participants  could  only 
use  the  target  deck  flip  book  and  the  head-level  camera  display  to  complete  tasks.  Information  on  the  two  HDDs 
presenting  subsystem  information  was  also  not  needed  for  completion  of  the  tasks.  However,  this  information  was  not 
covered,  to  add  visual  complexity  to  the  workstation.  Participants  used  the  center  keyboard  for  data  entry.  To  change 
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the  camera’s  field  of  view,  participants  manipulated  the  right  and  left  joysticks  to  control  the  camera’s  orientation  and 
zoom  factor,  respectively.  The  simulation  was  hosted  on  six  Pentium  PCs. 


Figure  3:  UAV  sensor  operator  workstation  simulator. 


3.3  Experimental  procedures: 

Participants  were  given  an  overview  of  the  experiment.  They  were  told  that  the  goals  of  the  experiment  were  to:  1) 
gather  opinions  on  whether  the  synthetic  symbology  presented  helped  completion  of  the  tasks,  and  2)  evaluate  the 
appearance/behavior  of  the  synthetic  symbology,  as  determined  by  “characteristics”  the  experimenter  manipulated  in  the 
generation  of  the  symbology.  Participants  were  not  briefed  that  the  update  rate  would  be  varied  and  all  trial  instructions 
were  identical,  regardless  of  what  update  rate  was  in  effect.  Participants  practiced  each  task  separately  until  performance 
stabilized.  Each  participant  spent  4-6  hours  completing  the  training  and  experimental  trials. 

3.4  Experimental  tasks 

There  was  no  flight  control  task;  the  UAV  automatically  loitered  in  a  racetrack  pattern.  Participants  completed  four 
types  of  sensor  operator  (SO)  tasks,  described  below.  Only  the  Target  Marking  task  and  data  will  be  addressed  in  the 
present  paper. 

1  •  Target  marking:  center  cross  hairs  and  designate  (press  joystick  button)  as  many  unmarked  tanks  as  possible  in 
90  seconds.  If  an  unexpected,  high-priority  target  (fuel  truck)  is  spotted,  designate  it  as  well. 

2.  Large  area  search:  center  cross  hairs  over  landmark  instructed  to  find  and  press  joystick  button.  Performance 
was  examined  both  with  “PIP-On”  and  “PIP-Off  ’  conditions. 

3.  Route  tracking:  center  cross  hairs  on  instructed  route  and  move  crosshairs  along  route  highlighted  by 
symbology. 

4.  Data  entry:  after  designating  landmark,  using  keyboard,  enter  numeric  information  displayed  on  flag  that 
appears. 

The  number  of  trials  completed  with  each  of  the  4  SO  tasks  is  shown  in  Table  1 .  A  total  of  42  experimental  trials  were 
conducted  with  each  participant. 
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TABLE  1.  Number  of  Trials  with  Each  SO  Task  Type 


Type 

#of 

Trials 

SO  Task  Type 

1 

7 

Target  Marking/No  Unexpected  Target  Present 

7 

Target  Marking/Unexpected  Target  Present 

2 

7 

Large  Area  Search: 

PIP-On 

7 

Large  Area  Search: 

PIP-Off 

3 

7 

Route  Tracking 

4 

7 

Data  Entry 

3.5  Experimental  design 

A  within-subjects  design  was  utilized.  All  fourteen  male  participants  (average  age=24.7  years)  completed  trials  with 
each  of  seven  Telemetry  Data  Rates.  Performance  on  four  types  of  SO  tasks  were  evaluated  with  each  update  rate  (target 
marking,  large  area  search,  route  tracking,  and  data  entry;  Table  1).  For  the  large  area  search  task,  performance  was 
examined  both  with  PIP-On  and  PIP-Off.  The  42  trials  were  blocked  by  SO  task  type:  one  of  the  four  task  types  was 
examined  before  each  of  the  other  SO  task  types.  The  order  of  the  four  SO  task  types  was  counterbalanced  such  that 
each  of  the  four  task  types  followed  each  of  the  other  types  an  equal  number  of  times,  for  twelve  of  the  participants.  The 
order  was  randomized  for  the  remaining  two  participants. 

The  procedure  to  determine  the  order  of  trials  within  each  type  of  SO  task  differed  across  task  types.  For  the  SO  task 
to  be  reported  in  the  present  paper,  Target  Marking  Task,  a  total  of  14  trials  were  conducted  per  participant.  Two  trials 
were  conducted  (one  with  an  unexpected  target  and  one  without  an  unexpected  target)  with  each  of  the  7  update  rates. 

The  order  of  the  14  trials  was  randomized  independently  for  each  of  the  14  participants,  with  several  constraints:  a)  the 
two  trials  with  each  update  rate  were  separated  by  at  least  two  trials  with  different  update  rates  and  b)  either  type  of  target 
marking  trial  (unexpected  or  no  unexpected  target)  could  not  occur  more  than  3  times  in  a  row.  This  procedure  helped 
minimize  the  possibility  that  participants  developed  expectancy  as  to  which  trials  had  an  unexpected  target.  With  this 
procedure,  the  order  in  which  the  update  rates  were  experienced  was  randomized.  An  adaptation  period  preceded  each 
trial  to  familiarize  participants  with  the  appearance/behavior  of  the  symbology  (determined  by  the  update  rate  in  effect). 

A  total  of  7  target  sets  were  used.  The  assignment  of  target  set  number  to  update  rate  was  randomized  independently 
for  each  participant  with  two  constraints:  a)  a  given  target  set  was  used  twice  with  each  update  rate  (once  with  and  once 
without  an  unexpected  target)  per  participant,  and  b)  each  target  set  was  used  an  equal  number  of  times  (twice)  with  each 
update  rate  across  participants. 

3.6  Target  Marking  Task 

The  participants’  task  was  to  manually  control  the  camera  orientation  to  locate  and  designate  as  many  tanks  as 
possible  in  a  90  second  period.  For  each  trial,  participants  hit  a  red  key  on  the  keyboard  to  auto  slew  the  camera  to  an 
“Adaptation  Area”.  Once  the  slew  stopped,  the  participants  pressed  the  “M”  key  to  gain  manual  control  of  the  camera. 
The  participants  were  trained  to  then  move  the  camera  around  in  the  Adaptation  Area  to  become  familiar  with  the 
appearance/behavior  of  the  synthetic  symbology  under  the  “characteristics”  (i.e.,  update  rate)  in  effect  for  that  particular 
trial.  After  10  seconds,  the  camera  auto  slewed  to  the  starting  point  for  that  trial.  This  point  was  in  the  same  field-of- 
view  as  a  “Target  Marking  Area”  that  was  enclosed  by  a  green  synthetic  boundary  line  (area  size  approximately  0.1 
square  mile). 

The  imagery  in  the  Target  Marking  Area  (Figure  4)  contained  30  tanks  that  were  pre-marked  with  green  flags  labeled 
“Tank”  and  15  “target”  tanks  that  had  no  flags.  (Participants  were  not  briefed  on  the  number  of  non-target  or  target  tanks 
per  trial.)  The  target  tanks  were  randomly  dispersed  in  the  area  according  to  three  difficulty  levels,  based  on  the 
assumption  that  target  tanks  that  were  not  close  to  pre-marked  tanks  would  be  easier  to  find.  At  full-zoom  screen  size, 
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“hard”  target  tanks  were  located  2  cm  (as  measured  on  the  front  of  the  monitor)  from  two  pre-marked  tanks.  “Medium” 
and  “easy”  tank  levels  were  4  and  6  cm,  respectively.  The  placement  and  difficulty  was  randomized  and  kept  balanced 
such  that  there  were  an  equal  number  of  hard,  medium,  and  easy  target  tanks  in  each  trial.  For  trials  in  which  an 
unexpected,  high-priority  target  was  also  present,  an  image  of  a  fuel  truck  was  located  such  that  two  constraints  were 
met:  the  truck  was  within  5  degrees  field-of-view  from  the  visual  center  point  required  for  designation  of  one  of  the  15 
target  tanks,  and  b)  the  truck  was  either  on  or  just  outside  the  synthetic  boundary  line  denoting  the  Target  Marking  Area. 


Figure  4:  Sample  Target  Marking  Area  showing  simulated  imagery,  tanks,  and  fuel  truck  (unexpected  target), 
as  well  as  synthetic  overlay  (boundary  line  and  flags). 


Figure  5:  Illustration  of  tank  (left)  and  unexpected  high  priority  fuel  truck  (right)  at  one  orientation. 

When  slewing  stopped  in  the  same  field-of-view  as  the  Target  Marking  Area,  the  participants  again  pressed  the  “M” 
key  to  resume  manual  camera  control  and  adjusted  the  left  joystick  to  zoom  in  all  the  way  to  locate/designate  tanks. 
Participants  then  moved  the  right  joystick  to  control  camera  orientation  in  order  to  center  the  cross  hairs  on  each  target 
tank  and  press  the  designate  button  on  the  joystick.  Once  a  target  tank  was  designated,  a  green  flag  was  presented  at  the 
location  stating  “Tank”.  If  the  unexpected,  high-priority  truck  was  located,  participants  were  trained  to  designate  it  as 
rapidly  as  possible  and  then  resume  the  task  of  locating/designating  the  other  tanks  until  the  time  limit  expired. 
Participants  also  heard  an  auditory  “ding”  whenever  a  flag  appeared  as  a  result  of  hitting  the  designation  button  when  the 
cursor  was  centered  on  a  tank  or  truck.  If  the  designation  button  was  pressed  when  the  cursor  was  not  centered  on  a 
target,  then  a  flag  was  dropped,  but  an  auditory  cue  was  not  presented.  Participants  had  90  seconds  to  locate  and 
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designate  15  target  tanks  (and  1  truck,  if  present),  with  the  goal  of  only  designating  each  one  once.  When  the  time  limit 
expired,  the  simulation  froze  and  a  message  “Trial  Over”  was  displayed. 

After  each  trial  was  completed,  participants  completed  a  post-trial  questionnaire  that  involved  rating  the 
appearance/behavior  of  the  synthetic  symbology  for  that  trial,  as  well  as  the  degree  to  which  it  was  distracting  and 
impacted  task  performance.  There  were  also  scales  for  participants  to  rate  their  perceived  situation  awareness,  task 
difficulty,  task  performance  and  workload  for  that  trial.  After  completion  of  all  14  target  marking  trials,  another 
questionnaire  was  administered  which  asked  whether  the  participants’  strategy  changed  depending  on  the 
appearance/behavior  of  the  symbology  and  whether  they  had  suggestions  on  how  the  synthetic  symbology  could  be 
improved. 

3.7  Measures 

The  number  of  15  “target”  tanks  designated  per  trial  were  recorded,  as  well  as  the  number  of  these  marked  more  than 
once.  The  number  of  false  alarms  (when  participants  marked  one  or  more  of  the  pre-marked  30  tanks)  was  also  tallied. 
Recorded  “misses”  were  instances  when  the  participant  pushed  the  designation  button  and  the  cursor  was  not  centered  on 
a  tank.  Measures  pertaining  to  the  unexpected  fuel  truck  target  included  whether  or  not  it  was  designated. 

Participants  completed  a  questionnaire  with  seven  rating  scales  after  every  trial.  Four  of  the  scales  dealt  with  the 
participants’  overall  impression  of  that  particular  trial  in  terms  of  situation  awareness,  task  difficulty,  task  performance, 
and  workload.  The  three  other  rating  scales  on  the  post-trial  questionnaire  focused  specifically  on  the  synthetic 
symbology.  After  completion  of  all  the  trials,  participants  were  given  an  opportunity  to  make  comments  on  the  strategy 
they  used  in  the  target  marking  task  and  on  how  the  symbology  might  be  improved. 

4.0  RESULTS 

Both  the  performance  and  subjective  data  were  analyzed  to  determine  if  there  were  significant  differences  as  a 
function  of  update  rate.  Mean  measures  were  obtained  and  entered  into  an  analysis  of  variance  with  the  Huynh-Feldt 
epsilon  correction  applied.  Post  hoc  analysis  was  done  using  Bonferroni  pair-wise  comparisons. 

4.1  Performance  data 

Participants  completed  two  types  of  90-second  target  marking  trials:  target  marking  trials  that  had  1 5  unmarked  tanks 
and  no  unexpected  target,  and  target  marking  trials  that  had  one  unexpected  target  (track)  in  addition  to  the  15  unmarked 
tanks.  Table  2  summarizes  the  performance  data  results  for  these  two  types  of  trials. 

These  results  showed  that  the  target  marking  data  generally  did  not  significantly  differ  as  a  function  of  update  rate  for 
both  types  of  target  marking  trials.  To  obtain  a  larger  pool  of  data,  it  was  decided  to  combine  the  data  from  the  two 
target  marking  trials,  even  though  half  of  these  trials  had  one  additional  target  for  participants  to  find.  With  this  pooled 
data,  there  was  a  significant  effect  of  update  rate  on  the  number  of  targets  marked  (F(6, 78)=3.586,  £=.004;  see  Figure  6). 
Post-hoc  Bonferroni  paired  t-tests  showed  a  trend  (i(13)=3.587,  /j=0.070);  the  average  number  of  targets  marked  when 
the  update  rate  was  .5  Hz  (7.429  targets)  was  less  than  the  number  marked  with  the  24  Hz  in  effect  (9.000  targets).  It 
should  be  noted,  though,  that  the  average  number  of  targets  marked  across  the  seven  update  rates  only  varied  by  1.75 
targets,  with  the  number  of  targets  marked  with  .5  Hz  less  than  the  number  of  targets  marked  for  all  the  other  update 
rates. 

This  pooled  data  set  underwent  a  separate  analysis  to  examine  if  performance  as  a  function  of  update  rate  differed 
depending  on  the  target’s  difficulty  level  (easy,  medium,  or  hard,  determined  by  how  close  the  target  tanks  were  to  other 
pre-marked  tanks).  However,  the  results  still  showed  that  there  were  no  significant  performance  differences  as  a  function 
of  update  rate  (easy:  £=.949,  average=5.8;  medium:  £=.958,  average=5.52,  hard:  p=  .586,  average=5.30). 
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TABLE  2.  SUMMARY  OF  TARGET  MARKING  PERFORMANCE  DATA 


Trials  with 

Trials  with 

No  Unexpected  Target 

Unexpected  Target 

Number  of  Targets  Marked  (15  presented  in  each  trial 


8.49;  (7.29-9.07);  2.99. 


rr.nriaa  I  ttl 


Count  =  5 


Insufficient  data  for  analysis 


0.30;  (0.02-0.86);  0.56 


59 


5.60;  (4.39-7.39);  3.97 


8.82;  (7.57-9.64);  3.14 


72 


Average;  (Range);  Std.  Dev. 


Update  Rate 


Targets  Marked  More  Than  Once  _ 


Average;  (Range);  Std.  Dev. 


Update  Rate 


Number  of  False  Alarms* 


Average;  (Range);  Std.  Dev. 


Update  Rate 


Number  of  Misses** 


Average;  (Range);  Std.  Dev. 


Update  Rate 


Number  of  Unexpected  Targets  Marked  (1  presented  in  each  trial)* 


_ Average;  (Range);  Std.  Dev. 

Not  Applicable 

Update  Rate 


3  outliers  replaced  with  the  grand  mean. 

One  participant’s  data  removed  from  data  set  as  average  was  4.71  times  other  participants’  combined  average 


Count  =  1 


Insufficient  data  for  analysis 


0.25;  (0.09-0.64);  0.51 


5.67;  (4.00-7.31);  4.68 


F(6,72)=2.002,  p=.  125 


0.60;  (0.43-0.91);  0.46 


Figure  6:  Average  number  of  targets  marked  with  each  update  rate  for  all  target  marking  trials, 
both  with  and  without  unexpected  target. 
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4.2  Subjective  data 

Four  of  the  seven  rating  scales  (all  7-point)  dealt  with  the  participants’  overall  impression  of  that  particular  trial  in 
terms  of  situation  awareness,  task  difficulty,  task  performance,  and  workload.  For  all  factors,  the  ratings  significantly 
differed  as  a  function  of  update  rate  (see  Table  3). 


TABLE  3.  SUMMARY  OF  SUBJECTIVE  RATINGS  ON  TASK  COMPLETION 


Rating  Factor 

Update  Rate 

Post-hoc  Bonferroni  test 

Situation  Awareness  (SA) 

F(6,78)=4.264,  p<.  05 

Less  SA  with  1  Hz  than  10  Hz,  p<. 05 

Task  Difficulty  (Figure  7) 

F(6,78)=9.050,  p<.0\ 

.5  Hz  harder  than  24  Hz,  p<.0 1 
.5  Hz  harder  than  4,6,10  Hz,  p<05 

1  Hz  harder  than  24  Hz,  p<.05 

Task  Performance 

F(6,78)=3.314,  p<.01 

No  significant  differences 

Workload 

F(6,78)=4.884,  p<.01 

No  significant  differences 

Generally,  situation  awareness  ratings  increased  as  the  update  rate  increased.  However,  all  average  ratings  were 
greater  than  zero  on  the  rating  scale,  indicating  that  regardless  of  update  rate,  participants  thought  that  they  had  situation 
awareness. 

Average  ratings  for  task  difficulty  (Figure  7)  indicated  participants  perceived  the  target  marking  task  harder  with  low 
update  rates  (.5  and  1  Hz)  compared  to  the  task  with  high  update  rates  (4,  6,  10,  and  24  Hz).  The  ratings  for  the  two 
other  factors  showed  similar  trends:  participants  tended  to  rate  their  task  performance  higher  and  their  workload  lower 
for  trials  that  had  a  higher  update  rate  in  effect. 


1.5 


3 
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”  J  1 _ I 

0.5  1  2  4  6  10  24 

Update  Rate 

Figure  7:  Participants’  average  rating  on  the  perceived  target  marking  task  difficulty  for  each  update  rate. 

The  three  other  rating  scales  on  the  post-trial  questionnaire  focused  specifically  on  the  synthetic  symbology.  For  all 
three  scales,  the  ratings  significantly  differed  as  a  function  of  update  rate.  These  results  are  summarized  in  Table  4. 


1  March  2006 


10 


TABLE  4.  SUMMARY  OF  SUBJECTIVE  RATINGS  ON  SYNTHETIC  SYMBOLOGY 


Rating  Factor 

Update  Rate 

Post-hoc  Bonferroni  test 

Appearance 
Figure  8 

F(6,78)=48.80,p<001 

All  comparisons,  p<.01 

Symbology  with  .5  Hz  more  degraded  than  all  other  rates 

1  Hz  worse  than  4,  6,  10,  24  Hz 

2  Hz  worse  than  6,  10,  24  Hz 

4  Hz  worse  than  10  and  24  Hz 

Distraction 

Figure  9 

F(6,78)=30.5 1 ,  pc.OOl 

.5  Hz  more  distracting  than  4,6,10,24  Hz  (p<.01) 

1  Hz  worse  than  4,6,10  Hz  (p<01)  &  24  Hz  (p<. 05) 

2  Hz  worse  than  6  &  10  Hz  (p<01) 

4  Hz  worse  than  10  Hz  (p<. 05) 

6  Hz  worse  than  10  Hz  ((p<  01) 

Performance  Impact 
Figure  10 

F(6,78)=19.48,  p<.001 

All  comparisons,  /K.01 

Performance  impacted  by  .5  Hz  more  than  4,  6,  10,  24  Hz 

1  Hz  worse  than  6,  10,  24  Hz 

2  Hz  worse  than  10,  24  Hz 

Figure  8  illustrates  the  data  from  the  scale  that  asked  participants  to  rate  to  what  extent  the  appearance  of  the  synthetic 
symbology  was  degraded  (jumpy,  delayed,  not  smooth,  etc.).  The  data  shown  in  Figure  9  is  from  the  scale  asking  to  what 
extent  the  appearance/behavior  of  the  synthetic  symbology  was  distracting  to  participants.  For  both  of  these  questions,  a 
5-point  scale  was  used.  Participants  were  to  choose  a  “not  at  all”  category  to  indicate  there  was  no  degradation  or 
distraction  during  that  trial.  For  trials  in  which  the  symbology  was  degraded  or  distracting,  the  other  four  choices 
enabled  participants  to  rate  the  severity  of  the  problem.  For  both  of  these  measures,  the  ratings  significantly  differed  as  a 
function  of  update  rate.  Generally,  the  slower  the  update  rate,  the  worse  the  ratings,  indicating  that  participants  found  the 
synthetic  symbology  degraded  and  distracting  at  the  lower  update  rates. 


Figure  8:  Participants’  average  rating  on  the  appearance  of  the  synthetic  symbology,  for  each  update  rate. 
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Figure  9:  Participants’  average  rating,  for  each  update  rate,  on  whether  the  synthetic  symbology  was  distracting. 

The  third  rating  scale  addressing  synthetic  symbology  asked  to  what  extent  the  synthetic  symbology  impacted  task 
performance.  This  question  was  included  because  even  if  the  participants  noticed  degradation  in  symbology  and/or 
found  it  distracting,  it  may  not  have  affected  task  performance,  in  their  view.  There  were  seven  rating  options. 
Participants  were  to  choose  the  middle  category  to  indicate  that  they  symbology  had  no  impact  on  task  performance  for 
that  trial.  Otherwise  they  were  to  choose  one  of  the  three  categories  to  the  left  or  right  to  indicate  the  degree  to  which  the 
synthetic  symbology  hurt  performance  or  aided  performance.  Participant  responses  indicated  that  the  .5  and  1  Hz  update 
rates  hurt  task  performance,  whereas  the  other  (higher)  update  rates  aided  performance,  the  higher  the  update  rate,  the 
more  beneficial  for  task  performance. 


Figure  10:  Participants’  average  rating  on  the  impact  of  the  synthetic  symbology  on  task  performance,  for  each  update  rate. 
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CONCLUSIONS 


Participants  performed  well  on  the  target  marking  task.  On  average  they  marked  over  half  of  the  15  unmarked  tanks 
and  usually  caught  the  unexpected  targets,  despite  the  trials’  short  90-second  time  limit.  There  were  few  instances  of 
marking  the  same  target  twice,  as  well  as  misses  and  false  alarms.  Furthermore,  this  level  of  performance  was  fairly 
consistent  across  update  rates.  Although  the  performance  data  did  not  show  a  decrement  as  a  function  of  update  rate,  the 
subjective  ratings  indicated  that  task  performance  was  better,  and  the  symbology  less  distracting,  the  higher  the  update 
rate.  Generally,  subjective  ratings  were  particularly  unfavorable  for  the  .5  andl  Hz  update  rate  conditions. 

The  fact  that  there  were  not  consistent  performance  differences  between  update  rates  may  reflect  human  adaptability, 
bringing  resources  to  bear  to  ensure  that  the  task  is  achieved  regardless  of  the  update  rate.  A  few  subjects  mentioned  that 
when  the  symbology  was  degraded  (making  the  flag/tank  correspondence  less  clear),  they  had  to  compare  the  pattern  of 
the  synthetic  symbology  with  the  pattern  of  tank  images,  to  discern  which  had  been  marked.  Another  participant 
commented  that  when  the  symbology  lagged  or  was  choppy,  more  concentration  was  required.  With  only  fourteen  90- 
second  target  marking  trials,  this  extra  effort  (e.g.,  concentration)  to  maintain  a  high  performance  level  did  not  have  to  be 
sustained  for  a  long  duration.  It  may  be  that  differences  across  update  rate  would  have  been  evident  with  more  numerous 
trials,  longer  sessions,  and/or  realistic  missions  where  a  variety  of  tasks  are  being  time-shared.  It  also  may  be  that 
performance  decrements  would  have  occurred  with  less  salient  targets  and/or  a  more  cluttered  scene. 

Given  that  the  participants’  subjective  ratings  were  consistent  -  symbology  at  high  updates  rates  was  less  degrading 
and  distracting  and  served  as  an  aid  to  task  performance,  the  conclusion  can  be  made  that  the  highest  telemetry  rate 
supported  by  the  UAV  system  should  be  employed  when  applying  a  synthetic  vision  system.  At  a  minimum,  it  is 
recommended  that  rates  of  2  Hz  or  higher,  be  utilized.  This  conclusion,  based  solely  on  the  present  target  marking  data, 
should  be  revisited,  once  the  results  for  the  other  three  tasks  examined  in  this  study  are  available.  It  may  be  that  another 
update  rate  will  be  identified  as  optimal  when  considering  multiple  types  of  SO  tasks. 

It  is  also  important  to  note  that  there  are  several  points  in  the  overall  system  that  can  contribute  to  both  time  delay  and 
registration  error.  For  instance,  the  quality  of  the  UAV  positional  data,  is  subject  to  quantization  error,  random  delays, 
and  basic  measurement  error,  besides  problems  introduced  by  the  telemetry  system.  If  registration  errors  are  systematic 
rather  than  variable,  operators  might  be  able  to  adapt.  Indeed,  that  is  one  research  question:  How  much  registration  error 
and  time  delay  is  tolerable  for  a  UAV  application  before  task  performance  degrades  substantially?  For  some  of  these,  it 
may  be  possible  to  provide  a  manual  intervention  whereby  the  operator  can  dynamically  recalibrate  the  correspondence 
of  the  synthetic  and  real  world.  It  also  may  be  possible  that  advances  in  prediction  algorithms  may  help  overcome  the 
limitations  of  imprecise  and  tardy  data  input  to  the  synthetic  vision  system. 

As  was  mentioned  in  the  introduction,  one  concern  with  a  synthetic  vision  overlay  is  the  possibility  that  the  operator’s 
attention  is  tunneled  such  that  key  information  is  missed.  The  present  experiment  did  not  include  trials  without  synthetic 
symbology  to  systematically  test  this  issue.  However,  the  results  showing  that  60%  of  the  unexpected  targets  were 
marked  suggest  that  the  synthetic  symbology  utilized  in  the  present  experiment,  for  this  task  paradigm,  did  not  result  in 
cognitive  tunneling. 

In  sum,  the  target  marking  task  data  from  this  experiment  examining  the  effect  of  telemetry  update  rate  on  several 
representative  sensor  operator  tasks  provides  further  demonstration  that  a  synthetic  vision  overlay  aids  performance, 
regardless  of  a  range  of  update  rates.  There  remain,  however,  numerous  questions  for  the  research  community  at  large  to 
address  to  ensure  optimized  integration  of  a  synthetic  vision  system  for  UAV  applications.  It  is  highly  recommended  that 
resources  be  devoted  to  examining  the  application  of  synthetic  vision  systems  to  UAVs,  as  potential  operational  benefits 
include  faster  target  prosecution,  more  targets  serviced,  and  reduced  potential  for  collateral  damage. 
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